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Plot 


Since  1 980,  Congress  has  appropriated  about  $2  billion  to  the  Department 
of  Defense’s  (DOU)  Manufacturing  Technology  (ManTech)  Program  in  an 
effort  to  improve  the  productivity  and  responsiveness  of  the  defense 
indusirial  base.  For  fiscal  year  1991 ,  Congress  appropriated  more  than 
double  the  amount  DOD  had  requested. 


In  response  to  your  request  that  we  determine  whether  dod  systematically 
tracks  benefits  from  the  ManTech  program,  we  reviewed  the  nature  and 
extent  of  the  Office  of  the  Secretary  of  Defense  (OSD)  efforts  to  ensure  that 
the  (1)  ManTech  program  is  being  effectively  implemented  and  (2)  benefits 
from  ManTech  projects  are  being  appropriately  compiled  and  reported. 


Background 


^  The  ManTech  program  provides  research  and  development  seed  money  to 
help  develop  advanced  manufacturing  processes,  techniques,  and 
equipment.  ManTech  funds  are  to  be  used  only  when  private  industry  has  not 
committed  funds  for  manufacturing  technology  on  a  timely  basis  in 
support  of  DOD  requirements  or  when  results  are  directed  at  the  industrial 
improvement  of  government  facilities.  The  intended  purpose  of  the 
program  is  to  lower  manufacturing  costs,  improve  manufacturing 
processes,  and  improve  product  quality  tiirough  the  incorporation  of 
results  into  a  defense-related  manufacturing  process. 


Several  mqjor  reviews  of  the  ManTech  program  have  concluded  that  OSD 
needed  to  be  more  actively  involved  in  directing  the  program  and 
measuring  its  progress.  These  reviews  indicated  that  stronger  OSD 
leadership  and  oversight  could  better  coordinate  the  way  the  program 
addresses  technologies  of  common  concern  to  the  military  services^  15y 
establishing  a  single  program  plan.  _ _ _ 


'  !n  this  I  eport,  “iitihtaiy  services"  refers  to  the  Departments  of  the  Air  Force,  Army,  and  Navy,  as  well 
as  the  Defense  Logistics  Agency  (DLA). 
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To  strengthen  OSD’s  oversight  and  policy  guidance  role,  Congress  put  a 
provision  in  the  National  Defense  Authorization  Act  for  Fiscal  Year  199;i 
that  requires  OSD  to  develop  and  implement  a  National  Defense 
Manufacturing  Technology  Plan  with  goals,  priorities,  and  approaches  for 
the  ManTech  program.  The  military  services  are  expected  to  support  the 
goals  and  objectives  in  the  OSD  plan.  Prior  to  fiscal  year  1991,  Congress 
appropriated  funds  to  the  services  and  the  Defense  Logistics  Agency  for 
the  program.  For  fiscal  year  1991 ,  Congress  appropriated  approximately 
$312  million  to  the  ManTech  program,  with  $50  million  going  directly  to  OSD 
to  implement  this  plan  and  support  individual  projects.  An  OSD  official  told 
us  that  almost  all  of  its  1991  funds  had  been  released  to  the  services  and 
Defense  Advanced  Research  Projects  Agency  for  various  projects.  For 
fiscal  year  1992,  Congress  appropriated  $280  million,  with  $100  million 
going  to  OSD. 

Figure  1  shows  the  funding  levels  for  the  ManTech  program  from  fiscal  year 
1980  to  the  present. 


Figure  1:  Manufacturing  Technology  Program  Funding  Summary 
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Results  in  Brief 


OSD  does  not  have  reasonable  assurances  that  the  ManTech  program  is  being 
effectively  implemented.  Long-standing  problems  with  the  progiam’s 
central  management  infor.mation  system  have  gone  uncorrected.  The 
services’  annual  repoits  to  OSD  describe  the  individual  ManTech  projects  but 
do  not  address  the  extent  to  which  the  program  goals,  priorities  and 
planned  approaches  are  being  carried  out.  The  cost  savings  or  financial 
benefits  being  attributed  to  ManTech  projects  are  not  reliable.  OSD  has  not 
established  a  methodologj'  for  assessing  the  program's  impact. 

Congress  required  OSD  to  establish  a  Manufacturing  Technology  Plan.  In 
response  to  the  legislative  mandate,  osd  indicated  ttiat  it  will  be  taking  a 
stronger  role  in  planning,  but  so  far  has  not  establisucd  guidance  that  will 
enable  it  to  measure  and  evaluate  program  effectiveness. 


OSD  Needs  to  Improve 
Reporting  of  Program 
Results 


As  part  of  its  efforts  to  oversee  the  ManTech  program,  osD  requires  the 
services  to  supply  detailed  project  information  to  its  central  data  base  and 
to  prepare  annual  reports.  Despite  long-standir  j;  problems  with  both  this 
data  base  and  the  annual  reports,  osd  has  done  Utile  to  resolve  the 
problems. 


The  data  base,  created  over  6  years  ago,  ww)  intended  to  be  ?  n.tmagement 
information  system  on  the  military  service-  ’  ManTci  li  programs,  osd  and 
service  officials  told  us  that  they  expee^f*  •  a  centra)  data  base  n>  be  useful 
for  planning,  coordinating,  and  providut',  a  perspective  on  hi  w  program 
funds  were  being  used.  However,  our  review  of  t!te  data  base ;  nd 
interviews  with  program  officials  indi(  «ted  that  neithet  osd  uiir  the 
services  have  used  information  stored  in  the  central  data  base  )'or  these 
purposes,  osd  does  not  routinely  obtein  rr-'inagemeni  infoinvition  reports 
that  would  illustrate  how  it  intended  to  ttse  ihe  data  base. 


Fundamental  problems  have  existed  v/ith  the  data  oUfse  frf  m  its  inception. 
One  of  the  more  serious  problems  has  been  that  the  servii  es  used  different 
interpretations  of  the  term  “pr(\ieo'  lo  organize  and  sub:  >ii  information. 
osd  defined  a  “ManTech  project"  as  an  individuaUy  manag  jd  investment  that 
delivers  a  unique  end  product  and  required  that  a  “ManTtcn  project”  was  to 
be  the  common  unit  associated  with  all  central  data  base  submissions. 
However,  we  found  that  the  servl  :es  have  been  using  the  term  “project” 
for  a  wide  range  of  activities— from  funding  a  unique  technit  al  effort  for  a 
definite  time  period  to  funding  research  centers  on  a  continuing  basis.  As  a 
result,  information  in  the  central  data  base  is  for  dissimilar  activities 
instead  of  common  units. 
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Another  central  data  base  problem  was  missing  data.  The  data  base  was 
designed  to  contain  1 03  data  elements,  including  1 4  benefits  data 
elements,  tc  record  data  over  the  life  of  each  project.  The  services  either 
did  not  collect  all  of  this  information  for  their  own  use  or  formatted  it 
differently  from  OSD’s  requirement.  Therefore,  the  information  that  osD 
required  was  not  always  readily  available  from  the  services. 

We  obtained  information  on  benefits  in  OSD’s  data  base  for  selected 
projects  from  each  of  the  military  services’  programs.  We  found  the  data 
base  contained  little  information  related  to  benefits  associated  with  ManTech 
projects.  The  benefits  data  that  the  services  did  provide  to  OSD  had  limited 
usefulness,  because  it  was  inconsistent  and  could  not  be  synopsized  to  give 
an  overview  of  benefits. 

OSD  officials  acknowledged  that  the  data  base  was  incomplete  and  not 
used,  but  they  have  done  little  to  resolve  the  deficiencies.  oSD  initiated  a 
working  group  of  sei-vice  officials  that  met  three  times  during  1989  and 
1990  to  discuss  these  problems,  but  the  group  was  not  able  to  resolve  the 
difficulties  with  data  submission  because  osD  had  not  decided  how  it  would 
use  the  data  base.  Consequently,  the  military  services  were  reluctant  to 
provide  the  data,  dla  and  Army  recently  suspended  sending  information  to 
the  data  base,  pending  osd’s  management  review. 

OSD  also  required  the  military  services  to  prepare  and  submit  annual 
reports  on  their  ManTech  programs  that  were  intended  to  serve  as  a 
management  tool  for  OSD  OSD  asked  the  services  to  provide  information  on 
projects  funded,  projects  completed,  or  projects  canceled  during  the  past 
year,  as  well  as  some  information  on  total  funding  and  significant 
accomplishments  during  the  past  five  years.  The  services  were  also  asked 
to  list  the  briefings  they  had  held  to  present  results  of  completed  projects. 

We  found  that  the  services’  annual  reports  complied  with  osd’s 
requirements  to  describe  individual  ManTecli  projects.  However,  ttie  reports 
did  not  address  the  extent  to  which  established  program  goals,  priorities, 
and  planned  approaches  were  being  carried  out  because  OSD  did  not 
require  this  information. 
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OSD  Lacks  a 
Systematic  Approach 
for  Measuring  the 
Impact  of  ManTech 
Projects 


OSD  has  not  established  a  systematic  approach  for  measuring  the  impact  of 
the  ManTech  program.  The  program  wes  created  to  respond  to  gaps  in 
industrial  base  responsiveness  to  DOD’s  military  requirements.^  However, 
OSD  has  not  established  a  methodology  for  assessing  whether  the  ManTech 
program  is  achieving  the  expected  results. 

Currently,  oSD  requires  the  military  services  to  identify  and  quantify 
benefits  but  provides  no  guidance  on  how  to  do  it.  This  has  resulted  in 
estimates  of  the  cost  savings  or  financial  benefits  from  individual  ManTech 
projects  that  are  difficult  to  substantiate.  The  military  services  have 
conducted  studies  that  have  emphasized  the  difficulty  in  obtaining  “hard 
evidence"  of  cost  savings  or  financial  benefits  attributable  to  ManTech 
projects.  For  example,  one  study  acknowledged  that  estimates  of  financial 
savings  from  various  sources  over  the  years  were  used  to  do  a  savings 
projection  for  the  Air  Force  F-16  program.  The  study  stated  that  this  was 
done  although  validating  savings  used  for  the  projection  wo  uld  be 
extremely  difficult,  if  not  impossible,  and  was  not  attempted.  Two  other 
studies  concluded,  after  reviewing  83  ManTech  projects,  that  it  was  difficult 
to  obtain  “hard  evidence”  of  cost  savings  or  financial  benefits  attributable 
to  the  projects.  A  common  conclusion  of  these  studies  was  that  benefits  are 
difficult  to  substantiate  because  estimated  benefits  cannot  be  reliably 
linked  to  the  ManTech  investment.  Compounding  the  problem,  contractors 
do  not  always  cooperate  in  providing  the  necessary  information  to  track 
and  report  on  financial  bcopfi^s.  Contractors  place  restrictions  on  using 
their  proprietary  informatio;*.  >  reporting  financial  benefits. 

Another  problem  in  computing  savings  is  determining  the  period  over 
which  the  savings  occur.  For  example,  the  Navy  claimed  that  its  ManTech 
program  has  already  saved  over  $1.2  billion,  with  additional  projected 
savings  of  $7.3  billion,  but  provided  no  time  frame  for  the  future  savings. 
The  Navy’s  projections  depend  on  successful  implementation  of  ManTech 
projects  that  have  not  been  completed. 

In  another  example,  the  Air  Force  stated  that  it  expected  savings  to  exceed 
$1  billion  from  a  new  engine  inspection  technology  it  funded  as  a  ManTech 
project,  but  provided  no  time  frame  for  when  the  benefits  would  be 


“The  following  is.  an  example  where  DLA  is.  funding  a  project  to  develop  a  manufacturing  capability  to 
produce  substitutes  for  obsolete  microcircuits  when  they  ure  needed  as  spare  parts.  DLA  must  mamtain 
an  inventory  of  microcircuits  to  support  and  maintain  vanous  weapon  systems.  However,  contractors 
often  discontinue  manufacturing  microcircuits  even  though  replacement  microcu-cuits  are  routmely 
needed  long  before  the  useful  life  of  a  system  ends  DLA's  ManTech  project  is  mtended  to  develop  a 
flexible  manufacturing  technology  that  would  be  of  assistance  to  mdustry  to  produce  many  different 
replacement  microcucuits  as  they  are  needed. 
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realized.  The  Assistant  Secretary  of  Defense  (Production  &  Logistics)  used 
this  example  in  congressional  testimony  in  April  1991  to  say  that  $1  billion 
has  been  saved.  However,  our  review  of  the  records  indicates  that  the 
benefits  were  projected  out  to  the  year  2000. 

Our  discussions  with  officials  at  all  levels  indicated  that  they  believed  that 
the  focus  on  estimating  financial  benefits  had  made  the  reporting  and 
evaluating  of  results  difficult.  Nevertheless,  the  military  semces  continue 
to  estimate  and  report  cost  savings  resulting  from  ManTech  projects.  osD 
continues  to  rely  on  the  services’  estimates  when  discussing  the  projects’ 
achievements  and  does  not  analyze  the  reasonableness  of  the  services’ 
estimates.  OSD  does  not  test  the  accuracy  or  reliability  of  data  supplied. 


There  have  been  long-standing  criticisms  of  the  ManTech  program.  For 
example,  over  the  years,  the  ManTech  program  had  been  criticized  for  (1) 
spreading  small  amounts  of  funds  across  many  diverse  projects,  (2) 
funding  projects  that  were  largely  unrelated  and  not  focused  on  particular 
technological  innovations,  and  (3)  failing  to  disseminate  benefits  of  ManTech 
projects  to  industry.  These  and  other  criticisms  led  to  the  1991  legislation 
that  required  OSD  to  establish  a  Manufacturing  Technology  Plan  intended 
to  increase  program  impact. 

The  Congress  required  osD  to  establish  a  Manufacturing  Technology  Plan 
that  could  result  in  a  restructuring  of  the  overall  program.  osD  established 
a  multiagency  task  force'^  to  assist  it  with  developing  a  plan.  However,  the 
plan  has  not  yet  been  completed. 

OSD  officials  are  addressing  some  of  the  criticisms.  In  fact,  an  OSD  official 
stated  in  recent  congressional  testimony  that  OSD  was  already 
implementing  the  plan  and  achieving  “positive  results.”  The  OSD  official 
stated  that  OSD  had  “come  a  long  way  in  clarifying  program  goals  and 
increasing  communications  among  (DOD’sj  manufacturing  community.” 

In  response  to  the  legislative  requirement,  OSD  recently  adopted  a  more 
focused  approach  that  would  better  coordinate  the  selection  of  projects. 
However,  our  review  indicates  that  osD  does  not  have  any  evaluation 
criteria  to  substantiate  or  corroborate  claims  of  positive  results. 

^riio  DOD  ManTech  Task  Force  was  composed  of  repieseiUativ  es  from  Army,  Navy,  Air  Force,  DLA, 
OSD,  'he  Defense  Advanced  Research  Projects  Agency,  the  Strategic  Defense  Initiative  Organization, 
the  Departments  of  Coimnerce  and  Energy,  ilie  National  Science  Foundation,  and  tlie  National 
Aeronautics  and  Space  Adiiunistratic  n 
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Recommenrifition 


As  an  integral  part  of  completing  the  legislatively  required  Manufacturing 
Technology  Plan,  we  recommend  that  the  Secretary  of  Defense  revise  the 
system  of  controls  designed  to  pro’dde  assiurances  that  the  ManTech 
program  is  being  effectively  implemented.  Such  a  revised  system  should 
include  guidance  to  ensure  that  the  military  services  routinely  and 
uniformly  report  on  (1)  the  extent  to  which  they  have  sound  rationales  to 
demonstrate  they  are  funding  projects  that  industry  would  not  fund  on  a 
timely  basis  (2)  the  results  of  the  projects  measured  against  standardized 
criteria,  and  (3)  their  progress  in  meeting  established  program  goals, 
priorities,  and  planned  approaches. 

The  Secretary  of  Defense  should  also  revise  ManTech  program  guidance  to 
demonstrate  how  the  military  services’  program  data  will  be  used  to 
evaluate  the  overall  ManTech  program. 


AflenCV  Coininents  and  commenting  on  a  draft  of  this  report,  DOD  agreed  that  the  management 
1  •  of  the  ManTech  program  could  be  strengthened  and  tlie  program’s  overall 

(Jut  revaluation  effectiveness  could  be  improved.  However,  DOD  stated  that  its  “program 

managers  cannot  be  expected  to  manage  effectively  when  a  third  to  a  half 
of  the  program  serves  only  congressional  constituent  special  interests." 

We  recognize  that  in  fiscal  year  1991  Congress  more  than  doubled  the 
amount  DOD  had  requested  and  directed  the  use  of  a  large  portion  of  the 
appropriated  funds.  However,  we  do  not  believe  this  alleviates  DOD  from  its 
responsibilities  for  assuring  that  the  MauTech  program  is  being  effectively 
implemented.  Many  of  the  problems  cited  in  this  report  have  been  imown 
and  reported  for  years. 

The  DOD  comments  are  presented  in  their  entirety  in  appendix  I. 


Scope  and 
Methodology 


To  review  program  oversight,  benefits  tracking  and  reporting  in  the 
ManTech  program,  we  made  site  visits  and  obtained  information  from 
program  officials  in  OSD  and  the  military  services.  We  visited  the 

Office  of  the  Deputj’  Chief  of  Staff  for  Concurrent  Engineering, 
Headquarters,  U.S.  Aimy  Material  Command,  Alexandria,  VA; 

Naval  Industrial  Resources  Support  Activity,  Naval  Base,  Philadelphia,  PA; 
Manufacturing  Technology  Directorate,  Wright  Research  and  Development 
Center,  Air  Force  Systems  Command,  Wright-Patierson  Air  Force  Base, 
OH; 
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•  Manufacturing  Engineering/Research  Office,  Defense  Logistics  Agency, 
Alexandria,  VA; 

•  U.S.  Army  Communications  and  Electronics  Command,  Fort  Monmouth, 
NJ;  and 

•  U.S.  Army  Armament,  Munitions,  and  Chemical  Command,  Picatinny 
Arsenal,  NJ. 

We  reviewed  regulations,  previous  reports  on  the  program,  studies  of 
ManTech  benefits,  program  management  documents,  and  studies  that 
addressed  issues  in  evaluating  research  programs.  We  reviewed  program 
data  bases  maintained  by  OSD  and  the  military  services  to  compare 
information  available  on  project  benefits.  We  obtained  copies  of 
documents  pertaining  to  the  design,  purpose,  and  use  of  the  central  data 
base.  We  obtained  printouts  of  data  the  services  told  us  they  maintained  to 
manage  their  programs  and  provide  data  to  osd,  copies  of  Military  Base 
Load  Status  Reports  that  identified  missing  data  in  the  services’ 
submissions,  and  printouts  of  benefits  data  from  the  central  data  base  for 
selected  cases  as  of  June  19, 1991. 

We  conducted  our  review  from  August  1990  to  September  1991  in 
accordance  with  generally  accepted  government  auditing  standards. 

We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense,  other 
interested  congressional  committees,  and  the  Director,  Office  of 
Management  and  Budget.  We  will  make  copies  available  to  others  upon 
request.  Please  contact  me  at  (202)  275-8400  if  you  or  your  staff  have  any 
questions  concerning  thi:;  report.  Other  m^or  contributors  to  this  report 
are  listed  in  appendix  1. 

Sincerely  yours, 


Paul  F.  Math 

Director,  Research,  Development, 
Acquisition,  and  Procurement  Issues 
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Appendix  I _ 

Comments  From  the  Department  of  Defense 


Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  comment  1 


See  comment  2 


ASSISTANT  SECRETARY  OF  DEFENSE 

WASHINOTON,  D.C.  tO)OI-tOOO 


mooucTioN  ANo  Fcbruary  13,  1992 

LOaiATICS 

Mr.  Frank  C.  Conahan 
Assistant  Comptroller  General 
National  Security  and 

International  Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Conahan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  draft  report,  "DEFENSE  INDUSTRIAL 
BASE:  DoD's  Manufacturing  Technology  Program  Needs  Systematic 
Evaluation,"  dated  December  23,  1991  (GAO  Code  396147)  OSD  Case  8923. 

The  Department  of  Defense  agrees  that  the  management  of  the  DoD 
Manufacturing  Technology  Program  can  be;  strengthened  to  improve  the 
overall  program  effectiveness.  The  DoD  is  taking  steps  to  do  just 
that.  Yet  the  report  gives  inadequate  consideration  to  those 
efforts.  Similarly,  the  Manufacturing  Technology  Program  has  been 
considered  a  successful  program  for  many  years.  That  also  is  not 
mentioned  in  the  report. 

In  addition,  and  even  more  important,  it  is  particularly 
troublesome  that  the  impact  of  congressionally  directed  programs  is 
not  mentioned.  The  DoD  program  managers  cannot  be  e.xpected  to  manage 
effectively  when  a  third  to  half  of  their  program  serves  only 
congressional  constituent  special  interests.  In  FY  1992,  for 
example,  fully  58  percent  of  all  Manufacturing  Technology  Program 
funds  appropriated  are  for  directed  programs.  None  of  these  programs 
have  met  any  of  the  criteria  cited  by  the  GAO  as  being  important  in 
the  project  selection  process.  None  of  these  programs  have 
quantified  the  projected  benefits,  either  in  dollar  terms  or  in  other 
measures,  nor  has  any  analysis  of  cost  effectiveness  been  performed. 
With  such  a  volume  of  directed  progra.ms,  it  is  inevitable  that  the 
G.AO  will  discover  problems  such  as  those  alleged  in  the  ...raft  report, 
and  then  blame  them  on  DoD  management  rather  than  on  the  source  of 
the  problem — the  congressionally  direc  ed  programs. 

Thus,  It  is  the  Department  view  that,  the  draft  report  lacks 
balance  and  fails  to  address  several  fundamental  issues.  The  result 
is  a  misleading  report.  To  reflect  the  true  situation  accurately, 
ttie  report  should  cite  the  positive  aspects  of  the  program,  the 
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dilenuna  of  the  directed  programs,  and  DoD  plans  to  Improve  areas  that 
need  strengthening. 

The  Department  has  spent  several  months  developing  a  National 
Defense  Manufacturing  Technology  Plan,  which  currently  is  in  the 
process  of  being  reviewed  by  the  various  DoD  Components.  The 
activities  associated  with  preparing  the  plan  have  addressed  all  of 
the  valid  concerns  expressed  in  the  draft  report. 

Specific  DoD  comments  on  each  of  the  report  findings  and 
recommendations  are  provided  in  the  enclosure.  The  Department 
appreciates  the  opportunity  to  comment  on  the  draft  report. 

Sincerely, 


Enclosure 


David  J.  Berteau 

Principal  Deputy  Assistant  Secretary  of 
Defense  (Production  and  Logistics) 
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GhO  DRMrr  REPOST— DATED  DECEMBER  23,  1991 
(GAO  CODE  396147)  OSD  CASE  8923 

DSnNSE  ZNDOSTRZAI.  BASE:  DOD  ZS  NOT  ADEQOATELY  OVERSEEZMG 
THE  MXNDrACTORZMG  TECHMOL06T  PROGRAM 

DEPARTMENT  OF  DEnNSE  COMMENTS 
***** 

rZNDZNGS 

rZNDZNG  A:  Tb*  M»oag«ctucino  Tschnoloov  Proarsa  Provides  Rsssarch 
and  B«vloBwnt  ‘‘S— d”  Monsv  to  Hslp  DsvIod  Ad«/«nesd  M>nuf»etu?ina 
Proessssi.  Tsehniauss.  The  GAO  explained  that 

Manufacturing  and  Technology  funds  are  to  be  used  only  when  private 
industry  has  not  committed  funds  for  manufacturing  technology  on  a 
timely  basis  in  support  of  DoD  requirements.  According  to  the  GAO, 
the  intended  purpose  of  the  program  is  to  (1)  lower  manufacturing 
costs,  (2)  improve  manufacturing  processes,  or  (3)  provide  product 
quality  through  the  incorporation  of  results  into  a  Defense-related 
manufacturing  process. 

The  GAO  noted  that  several  reviews  of  the  Manufacturing  Technology 
Program  (known  as  the  MauTech  Program)  have  concluded  that  the  Office 
of  the  Secretary  of  Defense  needed  to  be  involved  more  actively  in 
directing  the  program  and  measuring  its  progress.  The  GAO  pointed 
out  those  reviews  indicated  that  stronger  leadership  and  oversight  by 
the  Office  of  the  Secretary  of  Defense  could  better  coordinate  the 
way  the  program  addresses  technologies  of  common  concern  to  the 
Military  Services  by  establishing  a  single  program  plan. 

The  GAO  explained  that,  to  strengthen  the  oversight  and  policy 
guidance  rore  of  the  Office  of  the  Secretary  of  Defense,  the  Congress 
put  a  provision  in  the  National  Defense  Authorization  Act  for  FT 
1991,  which  requires  the  Department  to  develop  and  implement  a 
National  Defense  Manufacturing  Technology  Plan  with  goals, 
priorities,  and  approaches  for  the  ManTech  program.  The  GAO  observed 
that  the  Military  Services  are  expected  to  support  the  goals  and 
objectives  in  the  DoD  plan.  The  GAO  noted  that,  for  FT  1991,  the 
Co.ngress  appropriated  approximately  ?312  million  to  the  ManTech 
program,  with  ?50  million  going  directly  to  the  Office  of  the 
Secretary  of  Defense  to  develop  the  plan  and  support  individual 
projects.  The  GAO  reported  that  almost  all  of  the  FY  1991  funds  have 
been  released  to  the  Services  and  the  Defense  Advanced  Research 
Projects  Agency  for  various  projects.  The  GAO  also  pointed  out  that 
for  FY  1992,  the  Congress  appropriated  $280  million,  with  $100 
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million  going  directly  to  the  Office  of  the  Secretary  of  Defense. 

(pp.  1-4 /GAO  Draft  Report) 

POD  WESPONSE:  Partially  concur.  The  DoD  agrees  that  the  Office  of 
the  Secretary  of  Defense  needs  to  strengthen  its  program  oversight 
and  program  coordination.  However,  the  report  is  misleading  about 
the  DoD  ability  to  control  the  program.  The  GAO  omitted  an  important 
fact  in  its  discussion  of  congressional  program  add-ons  in  FY  1991 
and  FY  1992.  The  Congress  provided  additional  funds,  but  also  said: 
"Direct  them  to  this  specific  purpose" — generally  in  their  home 
districts.  In  FY  1991,  two  thirds  of  the  amount  added  or  almost  one 
third  of  the  total  amount  appropriated  was  directed  to  specific 
congressional  interests  ($103  million  out  of  $312  million  total) .  In 
1992,  the  prof.  ■)rtion  of  directed  efforts  is  even  larger.  Congress 
appropriated  $280  million  of  which  58%,  or  $163  million,  was 
"directed"  to  specific  Manufacturing  Technology  Program  projects. 

None  of  these  projects  have  been  evaluated  against  any  selection 
criteria,  no  benefits  have  been  quantified,  and  no  analysis  of  cost 
effectiveness  has  been  performed.  This  creates  an  environment  in 
which  it  impossible  to  be  an  effective  program  manager. 

It  is  significant  challenge,  at  best,  for  the  DoD  to  formulate  and 
execute  a  sound  Manufacturing  Technology  Program  of  such  scope  and 
complexity  just  on  the  merits  of  the  problems  that  need  to  be  solved. 
With  outside  interference  designed  to  support  "constituency 
interests,"  it  becomes  an  exercise  in  futility.  Resources  expended 
to  develop  program  plans  are  wasted.  Realistic  planning  and 
execution  of  sound  program  management  practices  are  impossible  in 
such  an  environment.  As  a  result,  the  DoD  is  returning  $163  million 
of  FY  1992  Manufacturing  Technology  Program  directed  efforts  to  the 
Congress  for  consideration  as  rescission  candidates. 

FINDING  B:  The  Office  of  the  Secretary  of  Defense  Laelca  Adequate 
Information  On  the  ManTeeh  Program.  The  GAO  reported  that  the  Office 
of  the  Secretary  of  Defense  has  required  the  Military  Services  to 
supply  detailed  project  information  for  a  central  data  base  and  to 
prepare  annual  reports.  The  GAO  concluded,  however,  that  despite 
long-standing  problems  with  the  central  data  base  and  the  annual 
reports,  the  Office  of  the  Secretary  Defense  has  done  little  to 
resolve  the  problems. 

The  GAO  noted  that  the  DoD  data  base,  created  over  6  years  ago,  was 
intended  to  be  a  management  information  system  on  the  Manufacturing 
Technology  programs  of  the  Military  Services.  The  GAO  reported  that 
the  central  data  base  was  expected  to  be  useful  for  planning, 
coordinating,  and  providing  a  perspective  on  how  programs  were  being 
used.  The  GAO  found,  however,  that  (1)  neither  the  Office  of  the 
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See  comment  6 


Secretary  of  Defense  or  t^4e  Services  have  'ised  the  information  in  the 
data  base,  (2)  the  Office  of  the  Secretary  of  Defense  does  not  obtain 
reports  from  the  system,  and  (3)  a  single  management  information 
report  has  never  been  generated  to  illustrate  the  intended  use  of  the 
data  base. 

The  GAO  concluded  that  fundamental  problems  have  existed  with  the 
data  base  from  its  inception.  ,e  GAO  asserted  that  one  of  the  more 
serious  problems  has  been  that  the  Services  use  different 
interpretations  of  the  term  "project"  to  organize  and  submit 
information.  The  GAO  found  that  the  Services  used  the  terra  "project" 
for  a  wide  range  of  activities — from  funding  a  unique  technical 
effort  for  a  definite  time  period  to  funding  research  centers  on  a 
continuing  basis.  Another  problem  the  GAO  found  was  that  the 
Services  did  not  collect  all  of  the  required  data  and,  as  a  result, 
the  data  were  not  always  readily  available. 

For  selected  projects  from  each  of  the  Military  Services  programs, 
the  GAO  obtained  information  on  benefits  from  the  DoD  data  base.  The 
GAO  found  that  the  data  base  contained  little  information  related  to 
benefits  associated  with  the  Manufacturing  Technology  projects.  The 
GAO  concluded  that,  because  it  was  inconsistent  and  could  not  be 
synopsized  to  give  an  overview  of  benefits,  the  benefits  data 
provided  by  the  Services  to  the  Office  of  the  Secretary  of  Defense 
had  limited  usefulness.  According  to  the  GAO,  officials  in  the 
Office  of  the  Secretary  of  Defense  acknowledged  that  the  data  base 
was  incomplete  and  not  used.  The  GAO  further  concluded  that  little 
has  been  done  to  resolve  the  deficiencies. 

The  GAO  reported  that  the  Office  of  the  Secretary  of  Defense  also 
required  the  Military  Services  to  prepare  and  submit  annual  reports 
containing  detailed  information  on  their  Manufacturing  Technology 
programs.  The  GAO  found  that  the  annual  reports  complied  with  the 
requirements;  however,  the  reports  did  not  provide  overall  summaries 
and  trends  associated  with  the  various  Service  programs.  The  GAO 
asserted,  for  example,  that  the  annual  reports  did  not  address  the 
extent  to  which  established  program  goals,  priorities,  and  planned 
approaches  were  being  carried  out.  (pp.  4-7 /GAO  Draft  Report) 

DOD  RESPONSE:  Partially  concur.  The  DoD  agrees  with  the  text  on 
pages  4-7  of  the  report,  but  contends  the  section  heading  is  not 
supported  by  the  text. 

The  GAO  apparently  failed  to  recognize  that  the  data  base  is  not  the 
only  source  of  information  on  the  Manufacturing  Technology  Program 
available  to  the  Office  of  the  Secretary  of  Defense.  The  DoD 
Components  routinely  provide  information  through  other  sources.  They 
fully  comply  with  the  data  requirements  of  the  Planning,  Programming, 
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and  Budgeting  Syateir.  During  the  Dofi  budget  cycle,  the  programs  are 
reviewed  through  the  chain  of  commaml  and  finally  approved  by  the 
Office  of  the  Secretary  of  Defense.  The  DoD  Components  also 
frequently  supily  information  on  an  ad  hoc  basis  for  a  wide  range  of 
information  requirements.  In  addition,  the  Components  also  prepare 
annual  plans — which  are  shared  with  the  Office  of  the  Secretary  of 
Defense. 

The  GAO  criticism  must  be  limited  to  the  effectiveness  of  the 
centralized  ManTech  data  case.  While  both  the  Office  of  the 
Secretary  ef  Defense  and  the  DoD  Components  recognize  the  data  base 
must  be  strengthened,  the  GAO  view  that  it  is  not  used  is  incorrect. 
The  data  base  (1)  serves  as  the  primary  information  source  of  current 
ManTech  program  information  to  the  DoD  Manufacturing  Technology 
Information  Analysis  Center,  which  serves  a  national  audience  and  is 
the  DoD  focal  point  for  manufacturing  technology  information,  (2) 
was  recently  used  to  prepare  part  of  the  input  to  position  papers 
used  to  establish  President  Bush's  position  on  the  need  to  continue 
machine  tool  voluntary  restraint  agreements,  and  (3)  is  used  to 
prepare  a  standard  handout  of  the  Office  of  the  Secretary  of  Defense 
which  describes  the  scope  of  the  Manufacturing  Technology  Program  by 
listing  the  processes  associated  with  more  than  1000  individual 
ir vestments  listed  in  the  data  base. 

Both  the  Office  of  the  Secretary  of  Defense  and  the  DoD  Components 
recognize  the  need  to  strengthen  the  centralized  database  to  make  it 
more  effective.  That  issue  was  discussed  on  numerous  occasions 
during  the  deliberations  of  the  Task  Force,  which  was  convened  to 
assist  in  the  preparation  of  the  National  Defense  Manufacturing 
Technology  Plan.  The  DoD  has  asked  the  Logistics  Management 
Institute  to  initiate  a  review  of  the  data  base  and  other  reporting 
requirements  in  an  atte.mpt  to  resolve  the  same  issues  identified  by 
the  GAO. 


FINPiyg  C:  The  Office  of  the  Secretary  of  Defense  Lacks  Axtoguata 
Criteria  for  Measuring  Impact  of  Manufacturino  Technology  Projects. 

The  GAO  observed  that  the  Office  of  the  Secretary  of  Defense  had  not 
established  adequate  criteria  for  measuring  whether  the  Manufacturing 
Technology  program  is  funding  the  appropriate  projects  and  the 
expected  results  are  being  achieved.  The  GAO  noted  that  the  Military 
Services  are  required  to  identify  and  quantify  benefits.  The  GAO 
pointed  out,  however,  the  Military  Services  have  conducted  several 
studies  that  have  emphasized  the  difficulty  in  obtaining  "hard 
evidence"  of  cost  savings  or  financial  benefits  attributable  to 
ManTech  projects.  The  GAO  found  a  common  conclusion  of  those  studies 
was  that  benefits  cannot  be  reliably  linked  to  the  ManTech 
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investment.  In  addition,  the  GAO  found  the  problem  is  compounded  by 
contractors  placing  restrictionc  on  using  their  proprietary 
information  in  reporting  financial  benefits. 

Another  problem  the  GAO  found  was  determining  the  period  of  time  in 
which  the  saving  occurred.  For  example,  the  GAO  stated  that  the  Air 
Force  expects  savings  to  exceed  one  billion  dollars  for  a  new  engine 
inspection  technology  it  funded  as  a  ManTech  project;  however,  no 
time  frame  is  provided  as  to  when  the  benefits  would  be  realized. 

The  GAO  found  the  records  indicated  that  these  benefits  were 
projected  out  to  the  year  2000. 

The  GAO  reported  that,  according  to  officials  at  all  levels,  the 
focus  on  estx.iating  financial  benefits  had  hampered  the  reporting  and 
evaluating  of  results.  Nevertheless,  the  GAO  found  that  (1)  the 
Military  Services  continue  to  estimate  and  report  cost  savings 
resulting  from  Manufacturing  Technology  Projects,  (2)  the  Office  of 
the  Secretary  of  Defense  continues  to  rely  on  the  Services  estimates, 
and  (3)  the  Office  of  the  Secretary  of  Defense  does  not  test  the 
accuracy  or  reliability  of  data  supplied  by  the  Services,  (pp. 
7-9/GAO  Draft  Report) 

POD  MSPOWSl!  Partially  concur.  The  GAO  is  incorrect  in  stating 
that  the  DoD  does  not  have  criteria  to  measure  the  effectiveness  of 
the  Manufacturing  Technology  Program.  Current  DoD  criteria  include 
(1)  cost  and  lead  time  reduction;  (2)  improved  product  quality  and 
reliability;  (3)  in^roved  safety;  (4)  reduced  pollution;  (5)  lower 
repair  costs;  and  (6)  providing  proven  manufacturing  options.  The 
DoD  and  GAO  the  differ  on  expectations  of  the  extent  to  which  those 
attributes  can  be  measured  in  dollar  tenns.  The  GAO  apparently  has 
concluded  that  the  DoD  should  be  able  to  quantify  each  of  these  cited 
attributes  in  monetary  terms,  and  to  then  compare  them  against  some 
standard  of  acceptance — which  would  then  indicate  where  they  fall  in 
some  spectrum  from  good  to  bad.  While  such  preciseness  is  certainly 
desirable,  it  may  be  impractical  or  not  cost  effective.  The  DoD 
recognizes  that  some  ManTech  projects  do  produce  direct  cost  savings. 
Others  may  produce  man-hour  reductions.  Where  possible,  the  DoD 
attempts  to  identify  those  benefits.  However,  in  some  cases  the  DoD 
does  not  expect  a  financial  benefit.  For  example,  a  safety  or 
pollution  investment  clearly  can  be  successful,  but  end  up  costing 
the  DoD  more  money  than  the  previous  processes.  In  those  cases,  the 
DoD  will  attempt  to  quantify  the  benefits  in  terms  of  safety  or 
pollution  control  measures. 

The  DoD  is  in  the  process  of  establishing  a  systematic  method  of  (1) 
objectively  evaluating  which  projects  are  most  appropriate  for 
funding  and  (2)  objectively  evaluating  completed  project  benefits. 
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The  DoD  intends  to  field  test  the  new  methodology  in  the  near  future.  , 


riNDINg  D:  9f  PtHtSif*  AttWPtgJiO 

itnhanee  Its  Planning  Hole.  Put  Does  Little  to  teprov  Proaria 
Ovraight .  The  GAO  reported  that  there  have  been  long-standing 
criticisms  of  the  Manufacturing  Technology  Program.  The  GAO  stated 
the  criticisms  led  to  the  1991  legislation,  which  required  the  Office 
of  the  Secretary  of  Defense  to  establish  a  Manufacturing  Technology 
Plan  intended  to  increase  program  impact.  The  GAO  explained  that  the 
Congress  required  the  Office  of  the  Secretary  of  Defense  to  establish 
a  Manufacturing  Technology  Plan  that  could  result  in  a  restructuring 
of  the  overall  program.  The  GAO  noted  that,  although  the  Office  of 
the  Secretary  of  Defense  established  a  multi-agency  task  force  to 
assist  in  developing  a  plan,  the  plan  has  not  yet  been  completed. 

The  GAO  reported  that,  according  to  some  officials,  they  are 
addressing  some  of  the  criticisms  and  that  the  plan  is  achieving 
"positive  results."  The  GAO  noted  that,  in  response  to  the 
legislative  requirement,  the  Office  of  the  Secretary  of  Defense 
recently  adopted  a  more  focused  approach,  designed  to  better 
coordinate  the  projects  to  be  selected.  The  GAO  concluded,  however, 
that  officials  seem  to  be  overstating  the  extent  of  their  progress. 
The  GAO  further  concluded  that  without  any  evaluation  criteria  to 
substantiate  or  corroborate  claims  of  positive  results,  the  Office  of 
the  Secretary  of  Defense  appears  to  be  substituting  program  advocacy 
for  program  evaluation,  (pp.  9-lO/GAO  Draft  Report) 

DOD  RESPOMSE:  Partially  concur.  Significant  effort  has  been 
expended  to  prepare  a  National  Defense  Manufacturing  Technology  Plan. 
It  is  currently  being  staffed  officially  by  the  Office  of  the 
Secretary  of  Defense  prior  to  being  released. 

Among  other  things,  the  Plan  embodies  an  integrated  approach  for 
strengthening  program  oversight.  Before  the  end  of  the  current 
fiscal  year,  the  Office  of  the  Secretary  of  Defense  will  implement  a 
Manufacturing  Technology  Program  Steering  Committee,  chaired  by  the 
Principal  Deputy  Assistant  Secretary  of  Defense  (Production  and 
Logistics)  and  a  Management  Committee  chaired  by  a  the  Director, 
Manufacturing  Modernization.  That,  coupled  with  a  restructuring  of 
the  Manufacturing  Technology  Advisory  Group  committees  membership, 
goals,  and  objectives,  provides  a  dual  management  improvement 
thrust — top  level  guidance  and  oversight  supported  by  well  organized 
field  level  planning,  coordination,  and  networlcing.  In  addition,  in 
order  to  assure  that  these  initiatives  are  effective,  the  Office  of 
the  Secretary  of  Defense  periodically  will  review  management,  fiscal, 
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and  technical  aspects  of  each  DoD  Components'  Manufacturing 
Technology  Programs. 

The  positive  results  referred  to  by  the  DoD  and  referenced  by  the  GAO 
referred  to  the  coordination  and  networking  that  has  taken  place 
among  the  highly  diverse  Task  Force  participants.  Various 
Manufacturing  Technology  Program  organizations  and  individuals  have 
coordinated  with  one  ano-her  to  a  greater  extent  than  ever  before. 


***** 

NECOMMENDATZONS 

MCOWiCMDATIOH  X  ;  The  GAO  recommended  that  the  Secretary  of  Defense 
establish  a  system  of  controls  to  provide  reasonable  assurances  that 
the  Manufacturing  Technology  Program  is  being  implemented 
effectively.  According  to  the  GAO,  such  a  system  include  guidance  to 
ensure  that  the  Military  Services  routinely  and  uniformly  report  on 
the  following: 

-  the  extent  to  which  the  Services  are  funding  projects  that 
industry  would  not  fund  on  a  timely  basis; 

-  the  results  of  the  projects  measured  against  standardized 
criteria;  and 

-  the  progress  in  meeting  established  program  goals, 
priorities  and  planned  approaches,  (pp.  lO-ll/GAO  Draft 
Report) 

PCO  RISPOMSl:  Partially  concur.  The  recommendation  implies  that 
either  no  systems  of  controls  exists,  or  that  the  current  system  does 
not  provide  reasonable  assurance  that  the  Manufacturing  Technology 
Progrant  is  being  implemented  effectively.  That  opinxcn  is 
inconsistent  with  the  facts.  It  certainly  does  not  reflect 
adequately  any  the  many  positive  aspects  of  the  Manufacturing 
Tsclujology  Program,  which  seemed  to  have  been  totally  ignored  in  the 
GAO  report.  Nevertheless,  there  is  no  question  that  the  management  of 
the  Manufacturing  Technology  Program  can  be  strengthened. 

It  is  pointed  out,  however,  that  even  if  DoD  best  efforts  were 
perfect,  it  would  be  impossible  to  manage  the  Manufacturing 
Technology  Pcrogram  effectively  and  efficiently  as  long  as  the 
Congress  continues  to  direct  specific  programs  that  are  only  of 
interest  to  a  unique  constituency — which  often  does  not  include  the 
DoD.  The  Congress  directed  over  a  third  of  the  total  amount 
appropriated  in  FY  1991  and  58%  of  the  amount  appropriated  in  FY 
1992.  The  DoD  cannot  be  expected  to  manage  any  program  effectively 
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Now  on  p.  7. 


with  that  type  of  external  forces  impacting  management,  nor  to  be 
able  to  measure  them  against  goals  and  criteria  that  played  not  role 
in  the  congressional  decision  to  direct  funding  to  specific  projects. 
While  many  DoD  ManTech  program  managers  are  highly  skilled, 
experienced  individuals,  they  cannot  follow  basic  program  policies, 
nor  do  rational  planning  in  such  an  environment.  The  GAO  cannot 
reasonably  expect  them  to  do  so. 

BlCOHaiHDATIOW  2:  The  GAO  recommended  that  the  Secretary  of  Defense 
revise  the  ManTech  program  guidance  to  demonstrate  how  the  Military 
Services  program  data  will  be  used  to  evaluate  the  overall  ManTech 
program,  (p.  11/GAO  Draft  Report) 

DOD  RgSPOWSt:  Concur.  During  the  past  year,  the  Department  had  a 
committee  working  on  a  revision  of  the  DoD  Manufacturing  Technology 
policy  document,  as  a  part  of  the  activities  necessary  to  prepare  the 
National  Defense  Manufacturing  Technology  Plan.  The  DoD  will 
continue  those  activities  and  expects  to  complete  the  revision  of  the 
policy  document  by  the  end  of  FY  1992. 
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The  following  are  GAO’s  comments  on  dod’s  letter  dated 
February  13, 1992. 


GAO  Comments 


1 .  Our  draft  report  pointed  out  that  OSD  established  a  multiagency  task 
force  to  assist  it  with  developing  the  congressionally  mandated 
Manufacturing  Technology  Plan.  We  pointed  out  that  the  plan  could  result 
in  a  restructuring  of  the  overall  program.  We  also  pointed  out  that  OSD  was 
conducting  a  management  review  of  tli^  central  data  base. 

2.  Congressional  direction  of  a  portion  of  this  program  does  not  alleviate 
DOD  from  its  responsibilities  for  assuring  that  the  ManTech  program  is  being 
effectively  managed. 

3.  DOD  has  not  yet  fully  addressed  many  of  the  valid  concerns  about  this 
program.  For  example,  dod’s  comments  acknowledge  that  it  needed  to 
strengthen  the  centralized  data  base  to  make  it  more  effective  and  had  not 
field  tested  the  new  methodology  for  determining  which  projects  were 
most  appropriate  for  funding  or  for  objectively  evaluating  completed 
project  benefits. 

4.  The  correct  title  of  gao's  report  is  the  one  shown  in  dod’s  letter  dated 
February  13, 1992:  “DEFENSE  INDUSTRIAL  BASE:  dod’s  Manufacturing 
Technology  Program  Needs  Systematic  Evaluation.” 

5.  We  modified  the  section  heading  to  better  reflect  the  text. 

6.  Our  draft  report  recognized  that  the  data  base  is  not  osd’s  only  source  of 
ManTech  information  and  pointed  out  that  DOD  required  the  military  services 
to  prepare  and  submit  annual  reports.  However,  our  review  of  these 
reports  also  indicated  that  they  do  not  contain  the  necessary  information  to 
evaluate  the  effectiveness  of  the  program  in  achieving  overall  goals  and 
objectives. 

We  did  collect  and  analyze  budget  and  planning  data  from  the  services  and 
OSD.  However  our  report  does  not  focus  on  this  data,  because  the  data  was 
not  intended  to  be  used  for  assessing  whether  the  ManTech  program 
achieved  the  expected  results. 

7.  ManTech  officials  in  OSD  and  the  semces  told  us  that  the  centralized  data 
base  was  not  used  in  managing  the  program  and  that  data  was  missing 
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because  long-standing  problems  with  data  submission  had  not  been 
resolved. 

8.  OSD’s  current  program  guidance  to  the  military  services  continues  to 
encourage  the  reporting  of  financial  benefits  on  a  project  basis,  but  does 
not  provide  criteria  for  reporting  either  financial  or  nonfmancial  benefits. 
Without  such  criteria,  there  is  no  reasonable  assurance  that  the  claimed 
savings  accurately  reflect  the  contribution  that  the  program  made  in 
achieving  the  benefits.  Our  review  suggests  that  the  claimed  savings  are 
probably  overstated  at  least  in  some  instances. 

9.  This  DOD  comment  seems  contradictory  to  its  earlier  comments.  For 
example,  dod  stated  that  at  best  it  is  a  significant  challenge  for  dod  to 
formulate  and  execute  a  sound  ManTech  program  just  on  the  merits  of  the 
problems  to  be  solved,  dod  indicates  that  because  of  outside  interference 
to  support  constituency  interests  planning  becomes  an  exercise  in  futility 
and  resources  expended  to  develop  plans  are  wasted,  dod  states  that 
realistic  planning  is  impossible  in  such  an  environment. 
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